An operating system operating on an electronic computer is designed with complex architecture. Therefore, the operating system has a vulnerability called a security hole. The security hole is a vulnerable structure due to a defect or a bug of the software design. By intruding inappropriately into the operating system, taking advantage of the security hole of the operating system, a malicious user may perform a malicious action such as hacking, cracking, etc. A method using a buffer-overflow occurring in a memory is used as the means for the malicious action.
Herein, the buffer-overflow is described. A program running on the electronic computer consists of two parts, which are a program code and program data. The program code is a read-only code written in a machine code. The program data is an information part, such as the memory address of the program code that is executed by the execution instructions of the operating system.
Generally, the program is stored in the hard disk of the electronic computer. The program is called up by the operating system and is executed. At this time, the entire or a part of the program is stored in RAM (Random Access Memory), which is the main memory of the electronic computer. The main memory is the memory that can read and write data directly from the CPU (Central Processing Unit) of the electronic computer. The main memory is managed by applying a memory address to each unit that stores data. The main memory is divided into a high memory address (High Memory) area and a low memory address (Low Memory) area. The high memory address area is an area that has addresses with small numbers. When the address number grows, it is defined as low memory address area. Hereafter, the main memory is described only as memory.
When the program is loaded by the operating system, a part of the memory is allocated for the program. The program data is stored in the high memory area of the allocated memory. And the program code is stored in the low memory area of the allocated memory. The program data is divided into the three data areas of stack data, heap data, and static data. These three data areas are arranged in the memory mutually independent of each other. When the program for execution is called up by the operating system from the hard disk, at first the program code is loaded and stored in the memory. The program data is continuously loaded and stored in the memory.
FIG. 5 is a schematic showing the architecture of the memory in the electronic computer. The high address side of the memory is a stack memory area. Whenever the program is executed, stack memories for the program and the subroutine of the program are reserved in the stack memory area. The stack memories are stored in order to the lower memory address side of the memory. The stack memory consists of an argument area, a return address area, a stack data area, etc.
The low memory address side of the memory consists of areas such as a heap data area, a static data area, a program code area, etc. The static data area is located more to the high memory address side than the program code area. The static data area is the memory area where global variables and static C++ class members are reserved. The heap data area is located more to the high memory address side than the static data area. The heap data area is the area allocated for malloc( ), alloc( ) functions of C language and for “new” operator of C++ language.
The stack memory area is accessed by the LIFO (Last-in, first-out) method. Function parameters e.g. the return address, local variables, etc. are stored in the stack memory. The return address indicates the address of the instruction, which follows the end of the executing instruction. This return address is the most important value.
FIG. 6 shows a program written in C language as an example. When the program is executed, the main( ) function is executed at first (Line 1 to Line 3), then sub(Data1) of the subroutine in Line 4 is called up, and the processing of the program is shifted to Line 10. In the sub( ) subroutine, the processing is shifted to the original address, which is called sub( ), by the return instruction in Line 16. At this time, it is explained how the data is stored in the memory.
As shown in FIG. 7, when the subroutine is called up, data is stored in the stack memory area. The “return address to main( )” is stored in the high memory address side of the stack memory area. Then, an area for storing local variables in the subroutine is reserved. In this example, variables i (in Line 11) and buf (in Line 12) are shown. The value of the argument Data of the subroutine is copied into buf by the strcpy instruction in Line 14. At this time, if the value of the Data argument is larger than the size of buf, the local variable i is overwritten. The return address to the main( ) routine is also overwritten only if the value is greater than the local variable i. Thus, if the data overflows the area, which is reserved for the data, of the memory, the overflowed data are stored in the area for another variable. This is the buffer-overflow. The return address is re-written to another value; therefore, the program can not operate normally.
Generally, when the program, the subroutine and the functions of the program are ended after the execution thereof, they are returned to the position indicated by the return addresses and the execution thereof is continued. However, as explained above, if the program has a defect in the structure, it becomes possible to overwrite the return address with the data overflowing the reserved area for the local variable during the writing of the data.
Generally, the program that produces the buffer-overflow due to re-writing the return address comes to be in an irregular running state. Depending on the circumstances, most programs get out of control or stop. When the return address returned to a memory address, which is intentionally prepared, the operating system continuously executes the instruction stored in this memory address without understanding that it is a false code. An attack taking advantage of the vulnerability caused by the buffer-overflow is the execution of the intentionally prepared code in this way.
The false code execution using the buffer-overflow cannot be grasped by the operating system, which is managing the execution of the program. To prevent this false code execution, it is very important that the means for preventing or to being able to detect the falsification or the re-writing of the return address.
Methods, for example, the following method 1 and method 2, are proposed to solve this problem. The method 1 is a method adding a correction to the operating system. The method 2 is a method making a mechanism for preventing the buffer-overflow in the compiler. The non-patent document 1 uses the method 1. In this project, to prevent the buffer-overflow of open source operating systems, the return address area of the stack is moved to another memory area where the function does not execute.
The non-patent document 2 uses the method 2. In this method, in the GCC compiler, the overwriting of data is detected by preparing a protection section in the lower memory area of the stack memory.
Moreover, in the case of patent document 1, a protection value area is defined in a memory device, and data of the stack memory is protected by preserving it in the protection value area, and the processing instruction is executed. Even if the processing instruction, such as a subroutine (and so on), is executed, the program counter is protected because the return address (and so on) is protected in the protection value area.    Non-patent document 1: Openwall Linux kernel patch project, URL: http://www.openwall.com/linux/    Non-patent document 2: StackGuard: Simple Stack Smash Protection go GCC, URL: http://www.immunix.com/˜wagle    Patent document 1: US20010013094 A1-2001-08-09, “Memory device, stack protection system, computer system, compiler, stack protection method, storage medium and program transmission apparatus”